Repository navigation
feat(tracing): stamp sgp evals run and row ids on SGP spans - #545
mohammadatallah-scale wants to merge 10 commits into
Conversation
Spans of a task whose task_metadata carries sgp_evals now get flat sgp_evals_generation_run_id, sgp_evals_row_id and sgp_evals_attempt_idx keys on the SGP copy. Sync and async agents read the task in the ACP server. Temporal agents carry it in the workflow memo and activity headers to the worker. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…val attrs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
P3 Refuted: continue-as-new keeps the memo (real dev-server run stamps the post-CAN run), |
Local activities go through start_local_activity, which the outbound interceptor did not override, so their spans carried no run or row id. Also pin the worker's interceptor wiring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry none Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…id cannot strip a queued span Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…load succeeds Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…o a span flood cannot evict them Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
andrewvo-scale
left a comment
There was a problem hiding this comment.
LGTM. The SDK is the right layer, since it is the only place that sees both the task metadata and the span. The keys it reads (generation_run_id, row_id, attempt_idx under the sgp_evals marker) match what sgp-evaluations writes in _unit_task_metadata. With no eval ids, _sgp_metadata returns exactly what base returns, so existing users see no change. All seven threads are resolved with fixes at head. On the first P1 (unregister on later events), keeping the unregister is right, because the server sends the full task, task_metadata included, on every call. Two non-blocking notes inline.
…ns without an SGP processor Child workflows inherit no memo and their workflow id is not the task id, so the parent hands both down in the child's memo and activities carry the task id in a header. Span pinning now starts only once an SGP processor exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both closed. Child workflows now inherit the ids and the task id (a789175), and |
| if not _capture_enabled: | ||
| return |
There was a problem hiding this comment.
If a plain span starts before an SGP processor exists, capture_for_span saves no empty marker. If an eval task later takes the same task ID, that earlier span can export with the eval task’s run and row IDs. It then appears in searches for work it did not do. Keep the empty result from the span’s start.
Knowledge Base Used: Testing and compatibility
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agentex/lib/core/tracing/sgp_evals.py
Line: 95-96
Comment:
**Plain spans get eval labels**
If a plain span starts before an SGP processor exists, `capture_for_span` saves no empty marker. If an eval task later takes the same task ID, that earlier span can export with the eval task’s run and row IDs. It then appears in searches for work it did not do. Keep the empty result from the span’s start.
**Knowledge Base Used:** [Testing and compatibility](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/testing-and-compatibility.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.|
@mohammadatallah-scale I re-checked a789175/8a170c1. The child-workflow memo + task-id header fix and skipping pins with no SGP processor both look right, and neither runs without eval ids. Two things before merge:
|
OVE-1253. Pairs with an sgp-evaluations PR that tags each generation-unit task.
The problem. Spans of an evals generation-unit task carry no run or row id, so a spans search cannot find them.
The fix.
trace_idandtask_idare both not the task id.Test plan
Changes since review
This PR is not ready to merge because an early plain span can be saved with a later eval run’s IDs.
Fix with agent prompt
Summary
Eval-marked tasks now add flat run, row, and attempt ids to their SGP span copies, making spans searchable by evaluation. The PR wires that attribution through ACP handlers and Temporal workers, while keeping queued spans tied to the task metadata they had when they started.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR Task[Eval task metadata] --> Memo[Workflow memo] Memo --> Header[Activity headers] Header --> Registry[Worker task IDs] Task --> Registry Registry --> Capture[Span-start capture] Capture --> Export[SGP span export] Registry -->|No capture saved| ExportReviews (6) · Last reviewed commit: "style: sort interceptor imports" · Reviewed by Greptile